Lecture 1

Probability basics

• Random phenomena are all around us: outcome of a coin flip, weather patterns, stock market, getting disease

• Random means non-deterministic, i.e. phenomena where there is an associated uncertainty about the outcome; we cannot predict perfectly (next day weather, roll of a die, price of Apple stock)

• Inability to predict perfectly could be due to lack of full information (e.g. flipping coin) or intrinsic (quantum mechanics)

• Error in measurements: anything we measure has uncertainty associated with it

• Randomness does not imply complete lack of pattern (e.g. a multiple coin toss will have about half heads and half tails), meteorologists can assess (and quantify!) whether the chance of rain is high or low

• Probability is the branch of mathematics that allows us to model randomness and studies properties of random phenomena

• Probability has application in computer science, machine learning, artificial intelligence, finance, statistics

• Probability allows as to model observations/data arising from phenomena/processes that are or can be modeled as random

• Statistics can be thought of as providing solutions to the inverse problem of probability: given observations/data infer properties about the underlying random phenomenon/process that generated the data

• Probability can be defined formally (at different levels of mathematical rigor) or can be treated more intuitively

Sample space

Definition: the set of all possible outcomes of an experiment of interest

Examples:

• Single coin tossing: $\mathrm{\Omega }=\{H,T\}$$\Omega = \{H, T\}$

• Month of birth of a randomly chosen person: $Omega=\{Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec\}$$Omega = \{Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec\}$

• Whether a Youtube video will be clicked when presented to a potential viewer $Omega=\{YES,NO\}$$Omega = \{YES, NO\}$

Event space

Event Space: all possible events (collection of outcomes) we will consider.

For discrete sample spaces event space is typically all possible subsets of $\mathrm{\Omega }$$\Omega$

• Example: single coin toss

  • Sample space: $\mathrm{\Omega }=\{H,T\}$$\Omega = \{H, T\}$

  • Event space: $\mathcal{F}=\{\mathrm{\varnothing },\{T\},\{H\},\{H,T\}\}$$\mathcal{F} = \{\emptyset, \{T\}, \{H\}, \{H, T\}\}$

• Example: birth month of a randomly chosen person

  • $\mathrm{\Omega }=\{Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec\}$$\Omega = \{Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec\}$

  • $\mathcal{F}=\{\mathrm{\varnothing },\{Jan\},\dots ,\{Dec\},\{Jan,Feb\},\dots ,\{Jan,\dots ,Dec\}\}$$\mathcal{F} = \{\emptyset, \{Jan\}, \dots, \{Dec\}, \{Jan, Feb\}, \dots, \{Jan, \dots, Dec\}\}$

• Q: how many elements in $\mathcal{F}$$\mathcal{F}$?

We require that union of events and intersection of events are also events:

$\mathcal{F}$$\mathcal{F}$ is closed under unions and intersections

E.g. $A=\text{First semester}=\{\text{Jan},\text{Feb},\text{Mar},\text{Apr},\text{May},\text{Jun}\}$$A = \text{First semester} = \{\text{Jan}, \text{Feb}, \text{Mar}, \text{Apr}, \text{May}, \text{Jun}\}$

$B=\{\text{May},\text{Jun},\text{July}\}$$B = \{\text{May}, \text{Jun}, \text{July}\}$

$A\text{ or }B=A\cup B=\{\text{Jan},\text{Feb},\text{Mar},\text{Apr},\text{May},\text{Jun},\text{Jul}\}$$A \text{ or } B = A \cup B = \{\text{Jan}, \text{Feb}, \text{Mar}, \text{Apr}, \text{May}, \text{Jun}, \text{Jul}\}$

$A\text{ and }B=A\cap B=\{\text{May},\text{Jun}\}$$A \text{ and } B = A \cap B = \{\text{May}, \text{Jun}\}$

Complements

$\mathcal{Z}$$\mathcal{Z}$ is closed under union, intersection, and set difference

Venn Diagrams:

Disjoint events

E.g.

• $A=\text{First semester}=\{Jan,Feb,Mar,Apr,May,Jun\}$$A = \text{First semester} = \{Jan, Feb, Mar, Apr, May, Jun\}$

• $B=\text{Second semester}=\{Jul,Aug,Sep,Oct,Nov,Dec\}$$B = \text{Second semester} = \{Jul, Aug, Sep, Oct, Nov, Dec\}$

DeMorgan's Laws

Example: Let $J$$J$ be the event "John is guilty" and $M$$M$ the event "Mary is guilty.”

• $(J\cap M{)}^c=\text{Not true that both John and Mary are guilty}$$(J \cap M)^{c} = \text{Not true that both John and Mary are guilty}$

• $J^c\cup M^c=\text{Either John or Mary are not guilty}$$J^{c} \cup M^{c} = \text{Either John or Mary are not guilty}$

• $(J\cup M{)}^c=\text{Not true that either John or Mary (or both) are guilty}$$(J \cup M)^{c} = \text{Not true that either John or Mary (or both) are guilty}$

• $J^c\cap M^c=\text{Neither John nor Mary are guilty}$$J^{c} \cap M^{c} = \text{Neither John nor Mary are guilty}$

(prove DeMorgan's Laws to practice with set operations)

Probability functions

A probability function is a 'set function' that assigns a real number to each event in $\mathcal{Z}$$\mathcal{Z}$:

2.

The probability reflects the chances an event occurs, 0 being impossible and 1 being certain

Example: Fair coin

Example: birth month of a randomly chosen person

Or perhaps a more reasonable assignment of probabilities would be proportional to the number of days in each month:

(This shows that it is us, the users who assign probabilities; probabilities are not 'laws of nature')

Any probability in a discrete sample space can be constructed like in the previous examples:

is a probability function.

Exercise: show this

Property 3 holds for any number of disjoint events:

E.g. the sets:

• $A=\{\text{First Trimester}\}=\{Jan,Feb,Mar\}$$A = \{\text{First Trimester}\} = \{Jan, Feb, Mar\}$

• $B=\{\text{Second Trimester}\}=\{Apr,May,Jun\}$$B = \{\text{Second Trimester}\} = \{Apr, May, Jun\}$

• $C=\{Nov\}$$C = \{Nov\}$

Are pairwise disjoint

In general:

Provided the events are pairwise disjoint:

Derived properties

If $A,B\in \mathcal{Z}$$A, B \in \mathcal{Z}$

(Good practice exercise to show these)

Repeated experiments/Product of sample spaces

E.g. Flip a coin twice:

E.g. Flip a coin $n$$n$ times:

E.g. Flip a coin and then pick a month at random:

Q: How many elements in $\mathrm{\Omega }$$\Omega$?

Repeated experiments/Product of sample spaces

If we have a probability function $P_1$$P_{1}$ defined in ${\mathrm{\Omega }}_1$$\Omega_{1}$ and a probability $P_2$$P_{2}$ defined in ${\mathrm{\Omega }}_2$$\Omega_{2}$ we can naturally defined a probability $P$$P$ in ${\mathrm{\Omega }}^1\times {\mathrm{\Omega }}^2$$\Omega^{1} \times \Omega^{2}$ as:

E.g.

(This is how we model independence, which will cover next week)

Uniform probability spaces

In many applications it makes sense to assign the same probability to all elements of a finite sample space

E.g. two coin flip:

In general, a uniform probability space with $|\mathrm{\Omega }|=n$$|\Omega|=n$, has:

And the probability of an event is the number of elements in the event divided by the total number of elements in the sample space:

E.g. Pick a single card from a well shuffled standard 52-card deck:

Multiplicative counting principle

Many problems in probability theory require that we count the number of ways that a particular event can occur. This kind of counting falls under the area of mathematics called combinatorics.

The Multiplicative counting principle (MP).

Suppose that we perform $r$$r$ experiments such that the $k^{\text{th}}$$k^{\text{th}}$ experiment has $n_k$$n_{k}$ possible outcomes, for $k=1,2,\dots ,r$$k=1,2, \dots, r$. Then there are a total of:

possible outcomes for the sequence of $r$$r$ experiments.

Example 1: Need to choose a password for an online account. Password must consist of two lowercase letters (a to z) followed by one capital letter (A to Z) followed by four digits (0,1, \dots, 9).

Example 2: How many subsets does a set with $n$$n$ elements have?

Permutations

How many five-card hands are possible from a standard fifty-two card deck? (if order matters)

In general, a $k\text{-permutation}$$k\text{-permutation}$ of $n$$n$ distinct objects is a way to arrange $k$$k$ objects out of the $n$$n$ in a row (order matters).

The number of $k\text{-permutations}$$k\text{-permutations}$, $p(n,k)$$p(n,k)$, is given by:

$n\text{-permutations}$$n\text{-permutations}$ are often refer to as just permutations. There are:

Combinations

How many five-card hands are possible from a standard fifty-two card deck? (if order does not matter)

Each arrangement was counted $5!$$5!$ times so the number of unordered arrangements is:

In general, a $k\text{-combination}$$k\text{-combination}$ of $n$$n$ distinct objects is a way to arrange $k$$k$ objects out of the $n$$n$ when order does not matter.

The number of $k\text{-combinations}$$k\text{-combinations}$, $c(n,k)$$c(n,k)$, is given by:

Card problem

Suppose we deal a 5-card hand from a regular 52-card deck. Which is larger, P(One king) or P(Two hearts)?

xxxxxxxxxx
4 * choose (48, 4) / choose (52, 5)
## [1] 0.2994736
choose(13, 2) * choose (39, 3) / choose (52, 5)
## [1] 0.2742797

De Mere's problem

• Dice game that played an important role in the historical development of probability.

• Chevalier de Méré had been betting that, in four rolls of a die, at least one six would turn up.

• He was winning consistently and, to get more people to play, he changed the game to bet that, in 24 rolls of two dice, a pair of sixes would turn up.

• De Méré lost with 24 and felt that 25 rolls were necessary to make the game favorable.

• Was De Méré right?

Simulating De Mere's problem

Single die roll in R:

xxxxxxxxxx
sample (1:6, size = 1, replace = TRUE)
[1] 2

Four rolls of one die

xxxxxxxxxx
sample (1:6, size = 4, replace = TRUE)
[1] 1 2 4 3

Simulating De Mere's problem

Checking if a six came up

xxxxxxxxxx
any(sample(1:6, size = 4, replace = TRUE) == 6)
## [1] FALSE

Full simulation:

xxxxxxxxxx
nreps = 1000
set.seed (2021)
results = numeric(0)
for (i in 1:nreps) results[i] = any (sample(1:6, size = 4, replace = TRUE) == 6)
mean(results)

## [1] 0.507

De Mere's problem

Questions:

1. Based on this simulation result, do you think the bet's favorable?

2. Derive/compute the actual probability (hint: use that all outcomes of the four rolls of a die are equally likely)

3. Simulate the second scheme (24 rolls of two dice). What can you say about the favorability of the bet?

4. Derive/compute the actual probability. How about for 25 rolls of two dice?

Birthday 'paradox'

How many people $n$$n$ do we need to have in a room to make it a favorable bet (probability of success greater than $1/2$$1/2$) that two people in the room will have the same birthday?

Assume all 365 b-days are equally likely.

1. Perform a simulation in R to answer this question (hint: use the base R function 'duplicate' to check that whether there are matching b-days)

2. Compute the probability by mathematical derivation and plot the probability as a function of $n$$n$. (Hint: use the multiplication principle to count)

Infinite (countable) sample spaces

E.g. flip a coin until first heads appears:

What is the right probability space for this experiment?

Sample space has to be infinite because no guarantee experiment will terminate in a finite number of steps!

If we assume that after $k$$k$ flips all $k\text{-tuples}$$k\text{-tuples}$ are equally likely what should the probability $P(k)$$P(k)$ be?

Does this result in a probability function?

Infinite (countable) sample spaces

For infinite sample spaces need to change additivity rule to countably additivity rule:

2'.

Verification of $P(\mathrm{\Omega })=1$$P(\Omega)=1$:

(Used that for a geometric series:

)

Q: What's the probability that it'll take an even number of tosses until the first heads?

Finite and countable sets: a mathematical aside

A set is finite if its elements can be put in one-to-one correspondence with with:

E.g., the set of students in the classroom, the set of inhabitants in the world, the set of stars in the Milky Way.

A set is countable if its elements can be put in one-to-one correspondence with with the natural numbers:

E.g. The set of natural numbers, the set of odd numbers $(n\to 2n+1)$$(n \rightarrow 2n+1)$, the set of even numbers $(n\to 2n)$$(n \rightarrow 2n)$, the set of primes, the set of rational numbers ($\mathbb{Q}$$\mathbb{Q}$)\textbf{!!}

Examples of infinite non-countable sets:

Lecture 2 - Conditional Probability and Independence

Review from Last Class

• Sample spaces (finite)

• Probability functions on discrete spaces

• Uniform probability spaces

• Multiplicative counting principle

• Permutations/combinations

Repeated Experiments/Product of Sample Spaces

E.g. Flip a coin twice:

E.g. Flip a coin $n$$n$ times:

E.g. Flip a coin and then pick a month at random:

Q: How many elements in $\mathrm{\Omega }$$\Omega$? (Answer: 24)

Product of Probability Spaces

If we have a probability function $P_1$$P_1$ defined in ${\mathrm{\Omega }}_1$$\Omega_1$ and a probability $P_2$$P_2$ defined in ${\mathrm{\Omega }}_2$$\Omega_2$ we can naturally define a probability $P$$P$ in ${\mathrm{\Omega }}_1\times {\mathrm{\Omega }}_2$$\Omega_1 \times \Omega_2$ as:

E.g.

This is how we model independence.

Infinite (Countable) Sample Spaces

E.g. number of coin flips until first heads appears:

What is the right probability space for this experiment?

Sample space has to be infinite because no guarantee experiment will terminate in a finite number of steps!

If we assume that after $k$$k$ flips all $k$$k$-tuples are equally likely what should the probability $P(k)$$P(k)$ be?

Does this result in a proper probability function?

For infinite sample spaces need to change additivity rule to countably additivity rule:

Verification of $P(\mathrm{\Omega })=1$$P(\Omega) = 1$:

(Used that for a geometric series: $1+r+r^2+\cdots =\sum _{k=0}^{\mathrm{\infty }}{r}^k=\frac{1}{1-r}$$1 + r + r^2 + \dots = \sum_{k=0}^{\infty} r^k = \frac{1}{1-r}$ if $0\le r<1$$0 \leq r < 1$)

By extension of the rule for finite sample spaces, the probability defined above is a proper probability function.

Conditional Probability

Example: You roll a fair 6-faced die. Let $A$$A$ be the event that the outcome is an odd number, $A=\{1,3,5\}$$A = \{1, 3, 5\}$. Let $B$$B$ be the event that the outcome is less than 4, $B=\{1,2,3\}$$B = \{1, 2, 3\}$. What is the probability of $A$$A$? What is the probability of $A$$A$ given $B$$B$?

We can write:

Definition

If $A$$A$ and $B$$B$ are events, and $P(B)>0$$P(B) > 0$ the conditional probability of $A$$A$ given $B$$B$ is defined as:

• Fraction of the probability $B$$B$ also in the event $A$$A$

• Tells us how to update probability in the presence of new information

Example:

• What is the probability that two cards drawn at random from a deck of playing cards will both be aces?

• What is the probability that two cards drawn at random from a deck of playing cards will both be aces if after dealing the first card it is an Ace?

Bayes Rule

From the definition we get the properties:

Multiplication rule:

Bayes rule:

###

Example: There are approximately 2.6 physicians per 1,000 people in the US (from world public health data by country)

Probability of choosing a physician if randomly choose a US inhabitant = $\frac{2.6}{1000}=0.0026$$\frac{2.6}{1000} = 0.0026$

• $A=\{\text{Being a Physician in the US}\}=\{\text{Physician}\}$$A = \{\text{Being a Physician in the US}\} = \{\text{Physician}\}$

• $B=\{\text{Being a Woman in the US}\}=\{\text{Woman}\}$$B = \{\text{Being a Woman in the US}\} = \{\text{Woman}\}$

• $P(\text{Physician})=\frac{2.6}{1000}$$P(\text{Physician}) = \frac{2.6}{1000}$

• $P(\text{Woman})=0.508$$P(\text{Woman}) = 0.508$ (From US Census)

• $P(\text{Woman}|\text{Physician})=0.36$$P(\text{Woman} | \text{Physician}) = 0.36$ (From Labor statistics)

Some Special Cases

• $A$$A$ and $B$$B$ disjoint $\Rightarrow P(A|B)=P(B|A)=0$$\Rightarrow P(A | B) = P(B | A) = 0$

• $B\subset A\Rightarrow P(A|B)=1$$B \subset A \Rightarrow P(A | B) = 1$

• $A\subset B\Rightarrow P(A|B)=\frac{P(A)}{P(B)}$$A \subset B \Rightarrow P(A | B) = \frac{P(A)}{P(B)}$

Conditional Probability is a Probability Function

For fixed $C$$C$ with $P(C)>0$$P(C) > 0$ the conditional probability $P_C(\cdot )=P(\cdot |C)$$P_C(\cdot) = P(\cdot | C)$ is a probability function:

1. $P_C(A)=P(A|C)\ge 0$$P_C(A) = P(A | C) \geq 0$

2. $P_C(\mathrm{\Omega })=P(\mathrm{\Omega }|C)=1$$P_C(\Omega) = P(\Omega | C) = 1$

3. $P_C(A\cup B)=P_C(A)+P_C(B)$$P_C(A \cup B) = P_C(A) + P_C(B)$ if $A\cap B=\mathrm{\varnothing }$$A \cap B = \emptyset$

Law of Total Probability

If the sample space can be partitioned as $\mathrm{\Omega }=\bigcup _{i=1}^n{A}_i$$\Omega = \bigcup_{i=1}^{n} A_i$, with $A_1,\dots ,A_n$$A_1, \dots, A_n$ disjoint, then:

(holds even for a countable partition)

In particular, for any event $A$$A$, the sample space can be partitioned as $\mathrm{\Omega }=A\cup A^c$$\Omega = A \cup A^c$:

Law oftotal probability example

The probability of infection from a certain virus upon exposure is 10% for children age < 13, 5% for ages 13-60, and 15% for ages 60+. What is the probability that a random individual is infected upon exposure in a population where $P(\text{Age}<13)=0.2$$P(\text{Age} < 13) = 0.2$, $P(13\le \text{Age}\le 60)=0.6$$P(13 \leq \text{Age} \leq 60) = 0.6$, $P(\text{Age}>60)=0.2$$P(\text{Age} > 60) = 0.2$?

Let $I$$I$ denote the event of infection:

Medical Testing Example

A diagnostic test has 99% sensitivity and 98% specificity.

If the population prevalence of the disease is 3%, what is the probability that an individual who tests positive is affected with the disease?

• $\text{Sensitivity}=P(\text{Test+}|\text{Affected})=0.99$$\text{Sensitivity} = P(\text{Test+} | \text{Affected}) = 0.99$

• $\text{Specificity}=P(\text{Test-}|\text{Not Affected})=1-P(\text{Test+}|\text{Not Affected})$$\text{Specificity} = P(\text{Test-} | \text{Not Affected}) = 1 - P(\text{Test+} | \text{Not Affected})$

• $(P(\text{Test+}|\text{Not Affected})=1-\text{Specificity}=0.02)$$(P(\text{Test+} | \text{Not Affected}) = 1 - \text{Specificity} = 0.02)$

• $\text{Prevalence}=P(\text{Affected})=0.03$$\text{Prevalence} = P(\text{Affected}) = 0.03$

• $P(\text{Affected}|\text{Test+})=?$$P(\text{Affected} | \text{Test+}) = ?$

• $P(\text{Test+})=P(\text{Test+}|\text{Affected})P(\text{Affected})+P(\text{Test+}|\text{Not Affected})P(\text{Not Affected})$$P(\text{Test+}) = P(\text{Test+} | \text{Affected}) P (\text{Affected}) + P(\text{Test+} | \text{Not Affected}) P (\text{Not Affected})$

• $=0.99\times 0.03+0.02\times 0.97=0.0297+0.0194=0.0491$$= 0.99 \times 0.03 + 0.02 \times 0.97 = 0.0297 + 0.0194 = 0.0491$

• $P(\text{Affected}|\text{Test+})=\frac{P(\text{Test+}|\text{Affected})P(\text{Affected})}{P(\text{Test+})}=\frac{0.0297}{0.0491}=0.605$$P(\text{Affected} | \text{Test+}) = \frac{P(\text{Test+} | \text{Affected})P(\text{Affected})}{P(\text{Test+})} = \frac{0.0297}{0.0491} = 0.605$

Monty Hall Problem

You're on a game show, and you're given the choice of three doors:

• Behind one door is a car

• Behind the others, goats

You pick a door, say No. 1, and the host, who knows what's behind the doors, opens another door, say No. 3, which has a goat.

He then says to you, "Do you want to switch to door No. 2 or keep prize behind door No. 1?"

Should you switch? Answer: Yes! Switching gives you 2/3 probability of winning

Independence

Two events $A$$A$ and $B$$B$ are independent iff (if and only if):

E.g. fair coin tossed twice:

The events $\{H\text{ in first toss}\}$$\{H \text{ in first toss}\}$ and $\{H\text{ in second toss}\}$$\{H \text{ in second toss}\}$ are independent.

(in fact any first toss outcome is independent of any second toss outcome)

Independence between $A$$A$ and $B$$B$ is equivalent to:

1. $P(A|B)=P(A)$$P(A | B) = P(A)$

2. $P(B|A)=P(B)$$P(B | A) = P(B)$

3. $A$$A$ and $B^c$$B^c$ are independent (or $A^c$$A^c$ and $B$$B$ are independent, or $A^c$$A^c$ and $B^c$$B^c$ are independent)

Independence of Multiple Events

$A_1,\dots ,A_n$$A_1, \dots, A_n$ are independent iff:

And also the equation above holds replacing any number of the $A_i$$A_i$s by their complements ($2^n$$2^n$ equations!)

Pairwise independence does not imply independence!!

Example: Two tosses of a fair coin

• $A=\{H\text{ in first toss}\}$$A = \{H \text{ in first toss}\}$

• $B=\{H\text{ in second toss}\}$$B = \{H \text{ in second toss}\}$

• $C=\{\text{two tosses are equal}\}$$C = \{\text{two tosses are equal}\}$

These are pairwise independent but not mutually independent

Lecture 3 - Discrete Random Variables

Review from Last Class

• Countable sample spaces (e.g. flipping a coin until first head, $\mathrm{\Omega }=\{1,2,\dots \}=\mathbb{N}$$\Omega = \{1, 2, \dots\} = \mathbb{N}$)

• Conditional Probability: $P(A|B)=\frac{P(A\cap B)}{P(B)}$$P(A | B) = \frac{P(A \cap B)}{P(B)}$

• Law of total probability: $P(B)=P(B|A_1)P(A_1)+\cdots +P(B|A_n)P(A_n)$$P(B) = P(B | A_1)P(A_1) + \dots + P(B | A_n)P(A_n)$ for a (disjoint) partition $\mathrm{\Omega }=A_1\cup \cdots \cup A_n$$\Omega = A_1 \cup \dots \cup A_n$

• Bayes theorem: $P(A|B)=\frac{P(B|A)P(A)}{P(B)}$$P(A | B) = \frac{P(B | A)P(A)}{P(B)}$

• Independence: $P(A\cap B)=P(A)P(B)$$P(A \cap B) = P(A)P(B)$ (or more intuitively $P(A|B)=P(A)$$P(A | B) = P(A)$)

• For $n>2$$n > 2$ events independence requires satisfying $2^n$$2^n$ equations

Q: Is disjoint the same as independent? No! Disjoint events cannot both occur, while independent events don't affect each other's probabilities.

Discrete Random Variables

A discrete random variable is a function $X:\mathrm{\Omega }\to \mathbb{R}$$X : \Omega \rightarrow \mathbb{R}$ that takes a finite or countable number of values $x_1,x_2,\dots$$x_1, x_2, \dots$

E.g. The number in the upper face of the rolled die, the sum of two dice

Notation:

• $\{\omega :X(\omega )=x\}=\{X=x\}$$\{\omega : X(\omega) = x\} = \{X = x\}$

• $P(\{X=x\})=P(X=x)$$P(\{X = x\}) = P(X = x)$

Probability Mass Function (pmf)

The pmf of a discrete random variable taking values $x_1,x_2,\dots$$x_1, x_2, \dots$ is the function:

If $X$$X$ takes on the values $x_1,x_2,\dots$$x_1, x_2, \dots$ then:

• $p(x_i)>0$$p(x_i) > 0$, and $p(x_1)+p(x_2)+\cdots =1$$p(x_1) + p(x_2) + \dots = 1$

• $p(x)=0$$p(x) = 0$ for all other $x$$x$

E.g. Fair coin flip:

Cumulative Distribution Function (cdf)

The cdf of a discrete random variable taking values $x_1,x_2,\dots$$x_1, x_2, \dots$ is the function:

If $X$$X$ takes on the values $x_1,x_2,\dots$$x_1, x_2, \dots$ then:

E.g. Fair coin flip:

Both the pmf and the cdf completely characterize all the probabilistic information about a random variable (two random variables can have the same pmf and cdf and be different).

Properties of the Cumulative Distribution Function

• $F(x)$$F(x)$ is increasing: $x_1\le x_2\Rightarrow F(x_1)\le F(x_2)$$x_1 \leq x_2 \Rightarrow F(x_1) \leq F(x_2)$

• $0\le F(x)\le 1$$0 \leq F(x) \leq 1$, $\underset{x\to -\mathrm{\infty }}{lim}F(x)=0$$\lim_{x \to -\infty} F(x) = 0$, $\underset{x\to +\mathrm{\infty }}{lim}F(x)=1$$\lim_{x \to +\infty} F(x) = 1$

• $F$$F$ is right continuous: $\underset{ϵ\downarrow 0}{lim}F(x+ϵ)=F(x)$$\lim_{\epsilon \downarrow 0} F(x + \epsilon) = F(x)$

Bernoulli Distribution

A random variable has a Bernoulli distribution

• Models experiments with only two possible outcomes

• E.g. coin toss (H vs. T), die comes up six (yes vs. no)

• A random variable with a Bernoulli distribution is called a Bernoulli trial

Binomial Distribution

$n$$n$ independent Bernoulli trials (e.g. flipping a coin $n$$n$ times): $X_1,\dots ,X_n\sim Bernoulli(p)$$X_1, \dots, X_n \sim Bernoulli(p)$

$X=X_1+\cdots +X_n$$X = X_1 + \dots + X_n$ counts the number of successes in $n$$n$ trials

• Models the number of successes in $n$$n$ trials

• For $n=1$$n = 1$ the Binomial distribution is the Bernoulli distribution

Example: Side Effects

Suppose it is known that 5% of adults who take a certain medication experience negative side effects. What is the probability that more than $k$$k$ patients in a random sample of 100 will experience negative side effects?

Binomial Distribution in R

pmf, cdf, and Random generation of a binomial random variable

xxxxxxxxxx
# pmf
dbinom(3, size=10, prob=0.3)
## [1] 0.2668279

# cdf
pbinom(3, size=10, prob=0.3)
## [1] 0.6496107

# Random generation
rbinom(n=1, size=10, prob=0.3)
## [1] 2

rbinom(n=3, size=10, prob=0.3)
## [1] 0 1 3

Side Effects Example (continued)

xxxxxxxxxx
1 - pbinom(1, size = 100, prob = 0.05)
## [1] 0.9629188

pbinom(1, size = 100, prob = 0.05, lower.tail = FALSE)
## [1] 0.9629188

pbinom(5, size = 100, prob = 0.05, lower.tail = FALSE)
## [1] 0.3840009

pbinom(15, size = 100, prob = 0.05, lower.tail = FALSE)
## [1] 3.705408e-05

Binomial distribution pmf

xxxxxxxxxx
par(mar=c(6,8,5,1))

plot(0:5, dbinom(0:5, size=5, prob=0.4), col='red4', type='p', pch=16, cex=1.3, xlab='x', ylab='p(x)', cex.lab=2, cex.axis=2)

Binomial distribution cdf

xxxxxxxxxx
par(mar=c(6,8,5,1))

plot(stepfun(0:5, c(0, pbinom(0:5, size=5, prob=0.4))), pch = 1, lwd=2, col='steelblue', xlab='x', ylab='F(x)', cex.lab=2, cex.axis=2, main='', verticals = F)

Simulating a binomial 3 different ways

Generating $n$$n$ Bernoulli trials

xxxxxxxxxx
Bernoulli_trials_1 = sample(0:1, 10, replace = TRUE, prob=c(0.3, 0.7))
Bernoulli_trials_1

## [1] 1 1 0 1 1 0 1 0 1 0

sum(Bernoulli_trials_1)

## [1] 6

Generating $n$$n$ Bernoulli trials using rbinom()

xxxxxxxxxx
Bernoulli_trials_2 = rbinom(10, size = 1, prob=0.3)
Bernoulli_trials_2

## [1] 0 1 0 0 0 0 0 0 0 0

sum(Bernoulli_trials_2)

## [1] 1

Directly sampling from the binomial

xxxxxxxxxx
rbinom(1, size=10, prob=0.3) 

## [1] 5

Geometric Distribution

A discrete random variable $X$$X$ has a geometric distribution with parameter $p$$p$, where $0<p\le 1$$0 < p \leq 1$, if its probability mass function is given by:

Models the (discrete) waiting time until an event happens. E.g. number of trials till first heads.

Example: Concert Ticket

You and a friend want to go to a concert, but there's only one ticket left. The salesperson decides to toss a coin until heads appears. In each toss heads appears with probability $p$$p$, where $0<p<1$$0 < p < 1$, independent of each of the previous tosses. If the number of tosses needed is odd, your friend is allowed to buy the ticket; otherwise you can buy it. Would you agree to this arrangement?

Geometric Memoryless Property

The probability it'll take $n$$n$ additional trials if the first $k$$k$ are failures is the same as the probability it'll take $n$$n$ trials at the beginning of the experiment.

Geometric Distribution in R

pmf, cdf, and Random generation of a binomial random variable

Warning: The definition of the geometric in R is the number of failures before the first success, i.e. $X-1$$X - 1$.

xxxxxxxxxx
dgeom(x=5, prob = 0.1)
## [1] 0.059049

pgeom(10, prob= 0.1)
## [1] 0.6861894

rgeom(n=1, prob= 0.1)
## [1] 0

rgeom(n=3, prob= 0.1)
## [1] 4 3 4

Geometric distribution pmf

xxxxxxxxxx
par(mar=c(6,8,5,1))
plot(0:50, dgeom(0:50, prob=0.1), col='red4', type='p', pch=16, cex=1, xlab='x', ylab='p(x)', cex.lab=2, cex.axis=2

Geometric distribution cdf

xxxxxxxxxx
par(mar=c(6,8,5,1))
plot(stepfun(0:50, c(0, pgeom(0:50, prob=0.1))), pch = 1, lwd=2, col='steelblue', xlab='x', ylab='F(x)', cex.lab=2, cex.axis=2, m